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Purpose of This Project 


The Every Student Succeeds Act (ESSA) gives states more power than they had under previous 
federal law to determine which schools are low-performing and gives school districts greater 
latitude to decide on strategies to improve these schools. The strategies chosen, however, must 
be backed up by evidence of effectiveness that fits into one of four “tiers,” ranging from strong 
evidence to evidence that demonstrates a rationale. While these requirements offer education 
leaders the flexibility to pursue innovative strategies, they present new and urgent 
implementation challenges and opportunities for state and local leaders. 


This report describes findings from the first phase of a project by the Center on Education 
Policy (CEP) designed to learn more about state and local implementation of ESSA’s evidence 
requirements and the roles of education leaders and researchers in advancing evidence use in 
school improvement. The findings reported here are from interviews conducted by CEP staff 
with state officials who oversee school improvement in seven states. The interviews focused on 
state experiences with initial implementation of the ESSA evidence requirements, including 
state efforts to identify evidence-based interventions and assist districts with their school 
improvement efforts. 


The seven states were purposefully selected to account for geographic diversity and included a 
mix of states with small and large populations and with stronger and weaker state control of 
education. The telephone interviews were conducted in February and March of 2019, using a 
semi-structured interview protocol. To elicit honest responses, we assured interviewees that 
their names or their states would not be identified. Readers should note that the findings in this 
report only reflect the experiences of officials in seven states and that other states may have 
different experiences and approaches to using evidence for school improvement. 


In later phases of the project, we plan to gather information about local implementation of the 
ESSA evidence requirements. We will also convene researchers, state and local education 
leaders, and other policy experts to promote dialogue across these sectors about the challenges 
of and promising approaches for using high-quality evidence for school improvement and 
related decisions. 


State and Local Roles in Meeting ESSA’s Evidence Requirements 


Under ESSA, state education agencies (SEAs) develop their own criteria, within broad federal 
parameters, to measure school performance and identify which schools in the state are the 
lowest-performing. In particular, states must identify three categories of schools in need of 
improvement: Comprehensive Support and Improvement (CSI) schools; Targeted Support and 
Improvement (TSI) schools; and Additional Targeted Support and Improvement (ATSI) schools. 
As described in Box A below, the CSI category is based on very low overall performance or 
poor high school graduation rates, while the TSI and ATSI categories are based on low 
performance of one or more student subgroups. 


Box A — Schools Identified for Improvement 


In school year 2018-19, the Every Student Succeeds Act requires states to identify three types of 
schools for improvement activities: 


e Comprehensive Support and Improvement (CSI) schools include a) the lowest performing 5% 
of schools in the state that receive federal Title I funds and b) high schools that fail to graduate at 
least two-thirds of their students. School districts and stakeholders work together to determine 
which actions to take to improve these schools, and the selected activity, strategy, or 
intervention must meet ESSA’s evidence requirements. The SEA approves the school 
improvement actions and monitors implementation. 


Targeted Support and Improvement (TSI) schools. In these schools a subgroup of students is 
consistently underperforming, as defined by the state. The school and stakeholders work 
together to determine which actions to take to improve these schools, and the selected activity, 
strategy, or intervention must meet ESSA’s evidence requirements. The school district approves 
the school improvement action and monitors its implementation. 


Additional Targeted Support and Improvement (ATSI) schools. In these schools the 
performance of any one subgroup of students on its own would lead to the school being 
identified as a CSI school. The school district approves the school improvement action and 
monitors its implementation. 


Source: Sections 1111 (d) and (c) of the Elementary and Secondary Education Act as amended by ESSA 


bo 


ESSA leaves it to local education leaders to decide which “activities, strategies, or interventions’ 
will be used to improve student learning at their lowest-performing schools; however, the 
chosen actions must meet one of four tiers of evidence described in Box B on page 3. 


States also play a role in helping school districts and schools implement ESSA’s evidence-based 
requirements. SEAs are expected to 1) provide technical assistance to districts; (2) approve the 
activities, strategies, or interventions to be implemented in CSI schools; (3) assist with funding 
through school improvement grants (SIGs); and (4) monitor implementation. 
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While the ESSA provisions are more flexible than the list of actions required for improving 
schools under the No Child Left Behind Act (NCLB), the new evidence requirements bring new 
challenges and responsibilities. State and local leaders will need to become more sophisticated 
consumers of evidence. They must be able to access and keep track of the steadily growing body 
of current education research and to discern the quality and objectivity of available evidence. 
They must also be prepared to categorize evidence according to ESSA’s tiers, which can be 
confusing and vague. Local leaders will need to determine which evidence-based interventions 
are most appropriate in the specific contexts of their struggling schools and navigate a sea of 
vendors that tout their products as “evidence-based.” Since state and local leaders cannot be 
expected to have expertise in all of these areas, some will likely need to partner with qualified 
and experienced research entities to assist with this process. 


Box B — ESSA Evidence Tiers 


Under ESSA, the activities, strategies, or interventions implemented in CSI, TSI, or ATSI schools 
must meet one of four tiers of evidence: 


e Tier 1. Strong evidence from at least one well-designed and well-implemented experimental 
study 


Tier 2. Moderate evidence from at least one well-designed and well-implemented quasi- 
experimental study 


Tier 3. Promising evidence from at least one well-designed and well-implemented 
correlational study with statistical controls for selection bias 


Tier 4. A rationale based on high-quality research findings or positive evaluation that such 
activity, strategy, or intervention is likely to improve student outcomes or other relevant 
outcomes; and includes ongoing efforts to examine the effects of such activity, strategy, or 
intervention. 


In districts or schools that receive SIGs from Title I funds, the chosen activities, strategies, or 
interventions must meet the evidence standards for Tiers 1, 2, or 3. 


Source: Section 8101(21)(A) of the Elementary and Secondary Education Act as amended by ESSA 


Main Themes from State Interviews 


States have adopted a wide variety of approaches to ESSA’s accountability and school 
improvement requirements. States are also at different stages of implementation: some states 
had already begun identifying schools for improvement using their ESSA accountability 
systems in 2017-18, and most started doing so in 2018-19. Given these variations, the 


1 Some states are waiting until 2019-20 to identify TSI schools for various reasons. One notable reason is to allow time 
for states with new assessments or accountability systems to accumulate multiple years of comparable student 
performance data. (Data from a previous test or different state standards would not be comparable.) Presumably, 
more than one year of comparable data is necessary to determine whether a student subgroup is “consistently” 
underperforming. 
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experiences of seven states are by no means representative. Still, several themes emerged across 
multiple state interviews that could be informative for other state leaders and researchers. These 
themes are summarized below and explained in more detail, with supporting quotations from 
state interviews, in later sections of the report. 


e Local education leaders, and even some state officials, are struggling with the ESSA 
evidence requirements, according to state interviewees. The greatest challenge is a lack 
of familiarity with research and methodology. As one state official observed, 
superintendents and principals are often hired for their managerial skills, not for their 
knowledge of school improvement research. 


e Some state leaders are concerned about their ability to help the larger-than-expected 
number of TSI and ATSI schools. Some state officials expressed surprise at how many 
schools have been identified as TSI and/or ATSI schools. These larger numbers are 
stretching the capacity of SEA staff and raising concerns about how to provide adequate 
assistance to schools. Most interviewees said they have the capacity to meet the letter of 
the law but would like to be able to do more for these schools and for CSI schools. Some 
are working with outside entities to expand their capacity. 


e States differ in the types of assistance they are providing to districts and CSI schools. 
Some states reported providing direct technical assistance and optional lists of 
interventions for CSI schools. Others have embedded SEA employees in CSI schools or 
are providing instructional coaches for identified schools. Some states are providing aid 
through regional education agencies. 


e States report that districts are reevaluating longstanding relationships with vendors 
of education products and services. According to state interviewees, the ESSA evidence 
requirements have prompted many state and district leaders to look more closely at 
vendors they have worked with in the past to ensure that their products and services are 
supported by evidence of their efficacy. 


e State views differed about the prohibition on using Title I SIG funding for tier 4 
interventions. ESSA requires states to reserve 7% of their federal Title I funds for school 
improvement purposes. At least 95% of this set-aside must go to school districts, 
consortia of districts, or education service agencies. However, these grant funds may be 
used only for activities, strategies, or interventions that meet the evidence criteria for 
tiers 1-3, but not tier 4. Some state interviewees objected to this policy on the grounds 
that tier 4 encouraged innovative, research-based strategies that had not yet been 
subjected to the more formal study methodologies required for higher tiers. Other state 
leaders favored the current prohibition on the grounds that it wasn’t appropriate to 
experiment with the most vulnerable children. 
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e Several state interviewees said the current U.S. Department of Education (ED) has not 
been helpful with ESSA implementation. Some noted that ED staff are slow to answer 
questions or had cancelled scheduled phone calls. However, one state official welcomed 
the hands-off approach because it has given the state and school districts the space to 
figure out ESSA and their new roles. 


e State leaders had several suggestions for how to make education research more 
accessible to educators. Examples included communicating research findings more 
clearly, eliminating barriers to accessing published research, inviting researchers to test 
their theories in schools, and other suggestions described later in this report. 


Interview Findings about Implementation of Evidence Requirements 


Our interviews were designed to gather information from SEA leaders in seven states about the 
implementation of ESSA’s evidence and school improvement provisions and related topics. 
Below we summarize findings from the interviews by topic, along with italicized quotations 
from the interviews that illustrate the findings and give a flavor of the voices of state leaders. 


Number of Identified Schools 


Most of the state officials that we interviewed were surprised by the number of schools 
identified for improvement under their ESSA accountability systems, especially the number of 
TSI/ATSI schools. The TSI/ATSI designation was most often prompted by the performance of 
the subgroup of students with disabilities, according to interviewees. 


We didn’t expect that student achievement would be that low among subgroups in so many 
schools. That was an awakening for all of us. 


Some of the seven states interviewed are annually identifying schools for improvement, while 
others are doing so every three years. One state decided to identify schools annually —and 
allow schools to exit this status after one year of improvement—based on comments made by 
educators and parents at the state’s stakeholder meetings on its ESSA plan: 


When we were having community meetings and focus group discussions to build our ESSA plan, 
one of the pieces of feedback we got, particularly from schools who had been identified for 
multiple years, is that over time, bearing the mantle of being a school identified for improvement 
has a self-defeating impact on staff, families, and the community ... We thought to address 
that, we wanted to identify the schools that are struggling, but make it easier for schools to get 
out of identification. 


SEA Capacity 


Most of the state officials interviewed noted that although their SEA staffs are “lean” and the 
ESSA evidence responsibilities are a “big lift,” their states could meet the letter of the law with 
current capacity — but they would like to be able to do more. Some officials reported that their 
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staff was stretched thin in providing assistance due to the higher-than-anticipated number of 
identified schools. At least one SEA was working with their state legislature to secure additional 
funding for this purpose. 


Right now, we think we are going to be able to do this [carry out school improvement 
responsibilities]. We know that there are going to be issues when we start trying to provide all 
the resources we need. Dollarwise, we definitely have concerns. | think for the past two years 
we’ve done a good job at being more involved and building up peoples’ capacity. 


Many of the states interviewed reported working with outside entities to extend their capacity, 
including federally funded regional centers, research organizations, nonprofit organizations, 
education associations, and universities. States are relying on these entities for help with 
different aspects of school improvement, such as identifying research-based interventions, 
determining the ESSA evidence level for lists of research-based interventions, or monitoring the 
implementation of interventions and gauging their effectiveness. 


One of the things we have in the pipeline is [a request for information] to help expand our 
evidence-based list. We are working with [a state university] to expand the list to include 
categories like early learning, literacy, equity, and social emotional learning. 


SEA Assistance to Districts with Low-Performing Schools 


The seven states participating in our interviews reported providing various forms of assistance 
to districts with schools identified for improvement. 


As required by ESSA, all of these states are using the school improvement set-aside to provide 
financial assistance to identified schools, but are taking diverse approaches to distributing it. 
States have the flexibility to distribute SIG funds through formula or competitive grants or, with 
district approval, to directly provide services to districts. One interviewee said their state is 
providing formula grants to schools but in varying amounts depending on the improvement 
category (CSI, TSI, ASTI). Other states are distributing these funds through a mixture of formula 
and competitive grants. Some states reported “keeping” the funds at the state level and using 
them to provide direct services to identified schools. Another state allocated SIG funds to the 
largest districts in the state and used the remaining set-aside funds to provide direct services to 
other districts. 


The interview states are using various approaches to help districts select school improvement 
strategies and comply with ESSA’s evidence requirements, as listed below. Some states 
emphasized note that their approaches were not new but were continuations or refinements of 
support structures developed prior to ESSA. In addition, officials in multiple states noted that 
state staff had already become familiar with ways to support identified schools through their 
experience with NCLB and subsequent federal waivers. 
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e State lists of pre-approved interventions. Several of the states interviewed were making 
available lists of interventions that were pre-approved by states as meeting evidence 
requirements, although districts are not limited to using those on the list. All but one 
said that the list was a non-exhaustive list, but one state indicated that districts were 
strongly encouraged to select from their list or get pre-approval for an approach that 
was not on the state list. Other states had decided not to make available this kind of list 
because they did not want to appear to endorse certain vendors or they did not think 
such a list could address all of their local contexts. 


In addition to the vendor list, we have put out guidance and technical assistance related to 
evidence-based practices... webinars, guidance documents ... We’ve given districts resources 
that they can look to as they are assessing evidence levels for prospective vendors. 


e Links to other lists and clearinghouses. In addition to providing lists of pre-approved 
interventions, many of the state officials we interviewed said their states are connecting 
districts with other resources that would help them identify appropriate interventions, 
such as the What Works Clearinghouse, ESSA Evidence, and ERIC. Some states are 
working with outside entities to align the information from these sources with ESSA’s 
evidence tiers. 


We put together a guidance document on how to use evidence-based practices. It walks districts 
through the problem-solving cycle of identifying their needs, selecting their interventions, 
implementation, and examining progress. There are a lot of links to external resources as well... 
Now we are building on this. 


e Presentations, webinars, technical assistance, and training. Interviewees reported that 
SEA staff have conducted statewide presentations and webinars and provided technical 
assistance on a range of topics, including the ESSA school improvement requirements 
and how to determine which interventions and strategies would be a good fit for an 
identified school. 


The primary approach that we are taking to help districts understand the evidence-based 
strategies requirements, as well as selecting evidence-based strategies, is first to do broad-based 
technical assistance and support. 


e Direct assistance. Several of the SEAs interviewed are providing direct state assistance 
to identified schools (mostly CSI schools) in the form of coaches or state-supported 
personnel embedded in schools. 


The coaches consult with [schools] on what interventions to use, putting the practices in place to 
make sure the interventions are being implemented successfully, and also making sure that they 
are evaluating programs and interventions that have already been put into place. 
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District Difficulties in Selecting Strategies 


We asked interviewees whether districts in their state were experiencing difficulties in choosing 
evidence-based school improvement strategies. Some officials said they had not heard of any 
district problems, but this was likely because districts were still involved in the selection 
process. (The interviews were conducted in February and March of 2019, and district school 
improvement plans were not due until later in the spring.) 


Some states officials did report difficulties with ESSA evidence requirements at the district 
level. Two state interviewees said these requirements had prompted uncomfortable 
conversations between districts and long-time vendors, as districts sought to ensure that 
vendors’ products and services met one of the ESSA evidence tiers. 


[There are] two big challenges. The first is, | like the vendor but | have no way of proving 
outcomes or if they are evidence-based. Challenge two, on the same side of the coin, | like this 
vendor and | see improvement but how do | capture that in a way that supports the ESSA tier of 
evidence. 


Another set of district challenges stems from the fact that district and school leaders may not 
have sufficient knowledge about research to access, interpret, and apply evidence to make 
informed decisions about interventions. As one interviewee explained, district and school 
leaders were “hired to be managers, not researchers.” Another state official described the 
challenges of moving from diagnosing why a school is low-performing to using research to 
determine an appropriate remedy: 


Schools are not generally made up of people who can read or consume research... Our schools 
are struggling with this, our advocates are struggling with this, the [SEA] is struggling with this . . 
. We are very good at collecting and looking at data, identifying needs based on data, 
hypothesizing root causes for why those needs exist. But where it begins to unravel is looking at 
the strategies, practices, programs, or problems to implement to address those root causes. 


One state official noted that although local officials like greater local control over school 
improvement in theory, they are finding it more difficult to exercise in practice: 


Overwhelmed is a dramatic understatement. Our leaders are really struggling with this... Part 
of it is our big push to increase local control. But they don’t really know what to do with the local 
control. It’s a situation where they say give us the control but tell us exactly what to do. We 
can’t exactly tell them what they want. 


Finally, one interviewee emphasized the importance of assessing the causes underlying a 
school’s low performance before selecting interventions. If districts have not correctly identified 
the root causes and are not monitoring a school’s progress, then it doesn’t matter which 
intervention they select. 
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Evidence Tiers 


ESSA allows strategies in evidence tier 4 to be used in any identified schools at the state’s 
discretion, except that schools receiving SIGs must choose from tiers 1 through 3. State 
interviewees report different state policies regarding tier 4. 


At least one of the states interviewed is limiting CSI schools to using strategies in tiers 1, 2, or 3 
only. (Very few strategies meet tier 1 criteria, state officials noted.) 


Another state is allowing districts to use tier 4 strategies in non-CSI schools, while making clear 
that districts opting for tier 4 must annually evaluate the chosen tier 4 strategy and be ready to 
drop it if it’s not working. Some states are encouraging districts to collect data on their 
implementation of tier 4 strategies, and a few states plan to review that data for possible 
additions to the list of research-based strategies. 


Only a handful of schools are taking the tier 4 route. We put out a piece of guidance around 
what tier 4 looks like, and | think that that guidance has scared a lot people away from doing tier 
4. It was written in such a way that only those with capacity could do it. 


One interviewee lamented that SIG funds cannot be used for tier 4 strategies because that 
official believes tier 4 is where innovation happens. 


Sometimes school leaders and schools find things that work through their own ingenuity and 
creativity and this limits that. There is no room to do something that is completely original 
because obviously there is no research base there. 


Other state officials expressed general concern about the rigidity of the categories: 


The categories are a double-edged sword. They encourage people to use more research-based 
interventions, but it could also lead to oversimplification of interventions. 


State Monitoring of District Interventions 


The officials interviewed described different approaches that their SEAS were undertaking to 
monitor implementation of interventions in low-performing schools. Some states plan to do this 
as part of their regular ESSA-required monitoring of district Title I implementation. One state 
that has embedded SEA personnel in low-performing schools are directing these personnel to 
file periodic reports. In other states, districts must enter information about their interventions 
into a database. 


What we do is have a robust monitoring process for schools in our state accountability system. . . 
One key component is not just [the state] looking at the data, but also an assessment of whether 
the conditions are in place for that school to sustain those efforts over time. We have a 
monitoring site visit process. 
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In addition, a few of the states interviewed are requiring SIG grantees to set aside a portion of 
their funding to evaluate their improvement efforts. 


Using Research 


All of the interviewees had suggestions for how researchers could address the needs and 
overcome the obstacles faced by local education leaders in accessing and using research. 
Suggestions included the following: 


e Produce plain-language summaries of research 


e Remove “paywalls” for scholarly journals to allow educators free or low-cost access to 
research 


There is a real “paywall” issue. We hope that researchers can make their research open access. 
That is important. 


e Label research studies with the appropriate ESSA evidence tier 


e Include references to the ESSA evidence tiers for research and strategies in the What 
Works and ERIC clearinghouses 


e Disseminate research more widely 


e Increase partnerships that bring researchers to work in schools to test their theories 


Indirect Consequences of Evidence Requirements 


Interviewees mentioned several indirect, incidental, or unintended consequences of ESSA’s 
evidence requirements. Some these are potentially positive, some may be problematic, and 
others have mixed implications. 


The evidence requirements, coupled with greater local decision-making authority, are spurring 
some districts to revisit their existing strategies to improve student achievement. This process 
can create uncertainty and disruption, as districts give greater scrutiny to arrangements with 
vendors or halt popular programs. But it also has encouraged state and district officials to make 
more informed choices, which could lead to better outcomes for students. 


Also on the positive side, the process of reviewing school improvement research and selecting 
strategies is itself a form of professional growth, as local education leaders and educators 
become more knowledgeable about appropriate strategies and more skilled in evaluating 
research. 


Instead of creating databases and lists of evidence, we really focused on upscaling our education 
leaders and our field staff and helping them get access to research and then make their own 
determinations based in the research. We really want to focus on getting people to understand 
what the research says and not what people are saying about the research. 
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State officials alluded to other unforeseen consequences that could be more problematic: 


e Limiting SIG funds to strategies in tiers 1, 2, or 3 may cause some districts to end 
popular programs, such as after-school programs, that were previously supported with 
these grants. 


e Interventions that meet the evidence tiers may not necessarily work in all school 
contexts. It may be very difficult for some school districts to find a research-based 
intervention that addresses the underlying reasons why a school was classified as low- 
performing and will work in their unique setting. 


State Observations about Other Aspects of ESSA 


During the course of the interviews, state officials raised broader issues about the role of the 
U.S. Department of Education and possible changes to ESSA. These are described below. 


U.S. Department of Education Support 


Most interviewees said their state was not receiving a great deal of support from the current 
U.S. Department of Education, although they had diverse views about this hands-off approach. 
One official wished that ED was more helpful and responsive. Another reported that scheduled 
phone calls with states have been cancelled and quarterly phone calls have not been scheduled 
at all. Two states mentioned the lack of written ESSA guidance from the current 
Administration, which had rescinded guidance formerly issued by the Obama Administration; 
one of these states reported using the Obama-era guidance to inform their ESSA decisions 
because the current ED is not helpful. 


The problem isn’t really with ESSA—it’s with the political upheaval between the department of 
ED, the Administration, and within the department of ED between career staff and political staff 
right now. We don’t feel like we are getting near the support from them. 


A different perspective came from a state official who said the lack of ED oversight has been a 
good thing because it has given the state and school districts space to figure out new 
approaches to assisting low-performing schools without federal interference. 


| think most SEAs would say the less we hear from the Department of Ed the better, at least 
overall. Anytime you get an email from them you kind of cringe. 
Changes to Federal Policy 


Finally, state officials suggested various changes to federal policies that would better promote 
the goal of school improvement in their view: 


e Give districts more flexibility to implement “home-grown” approaches to school 
improvement that meet tier 4 criteria. 
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e Overhaul federal clearinghouses to report information in terms of the ESSA evidence 
tiers. 


e Conduct a literature review through the U.S. Department of Education of all school 
improvement research. 


e Provide states with clear and consistent guidance about implementation requirements 
and parameters. 


In addition, one state proposed that a change that reflects the growing presence of for-profit 
providers in the education marketplace. That state suggested an amendment to ESSA that 
would: 


e Allow federal funds to be used only for interventions that are promoted or developed by 
nonprofit entities. 


It’s exactly like what happened to our standards. [For-profit vendors] are taking exactly what 
they sold prior to ESSA and recalibrating it. 


Conclusion 


State and local leaders in the seven states in which CEP conducted interviews appear to making 
serious efforts to implement ESSA’s evidence requirements. Although the requirements have 
presented some challenges, they appear to have increased awareness of the range of research- 
based strategies available to education leaders. State and local officials have also been asked to 
carefully examine the underlying evidence when selecting school improvement strategies, 
which can sometimes mean questioning or even changing longstanding relationships with 
vendors. Understanding research methods, weighing evidence, and selecting strategies that will 
work in specific contexts are all complex endeavors, so it is not surprising that state and local 
leaders sometimes feel challenged. But if these requirements encourage state and local leaders 
to become better consumers of education research, ESSA’s evidence requirements may benefit 
the field beyond the boundaries of the program. 


The research community could also take additional steps to make this process easier. Examples 
include pursuing additional partnerships with states and school districts and reporting research 
findings in ways that are accessible to non-researchers. Before conducting these interviews, CEP 
spoke extensively with both the Council of Chief State School Officers and AASA, The School 
Superintendents Association. Representatives of both organizations pointed out that states and 
districts need timely, relevant, and accessible research to support their goals and activities. They 
expressed a concern that too much research will render education leaders “data rich, but 
information poor.” By continuing to work hand in hand with researchers, state and local leaders 
can help shape the future of education research so it supports both policy and practice. 


Finally, CEP’s interviews demonstrate how the role of state departments has changed in recent 
years. The flexibility and authority given to state and local leaders under ESSA requires state 
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departments of education to play a bigger, more nuanced role in school improvement. It is 
understandable that some states feel overwhelmed or unprepared to take on these new 
responsibilities. All of the education groups CEP spoke with before conducting the state 
interviews addressed the issue of state and local capacity and how it may impact decision- 
making and implementation. Moving forward, the federal government and others will need to 
consider how they can better support the expanding role state departments of education play in 
school improvement. 
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